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Abstract 

We consider the ground state magnetic phase diagram of the two-dimensional Hubbard model 
with nearest and next-nearest neighbor hopping in terms of electronic density and interaction. 
We treat commensurate ferro- and antiferromagnetic, as well as incommensurate (spiral) magnetic 
phases. The first-order magnetic transitions with changing chemical potential, resulting in a phase 
separation (PS) in terms of density, are found between ferromagnetic, antiferromagnetic and spiral 
magnetic phases. We argue that the account of PS has a dramatic influence on the phase diagram 
in the vicinity of half-filling. The results imply possible interpretation of the unusual behavior of 
magnetic properties of one-layer cuprates in terms of PS between collinear and spiral magnetic 
phases. The relation of the results obtained to the magnetic properties of ruthenates is also 
discussed. 

PACS numbers: 
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I. INTRODUCTION 



Investigation of two-dimensional (2D) strongly correlated electronic systems attracts sub- 
stantial interest, which has been stimulated by the discovery of high-temperature supercon- 
ducting cuprates^. It is generally accepted that superconducting and magnetic properties of 
cuprates are closely related. While at half-filling cuprates are antiferromagnetically ordered, 
evolution of their magnetic properties with doping is an interesting challenge^. Neutron 
scattering in La2_pSrpCu04 reveals the coexistence of both commensurate and incommen- 
surate magnetic structures in the vicinity of half-filling (hole doping p < 0.02)- . At p ~ 0.02 
the system goes to an incommensurate (spin glass) state with the magnetic structure wave 
vector Q = {n — 6,71 — 6) (the corresponding long-range order is denoted in the following 
as diagonal); the incommensurability parameter 6 increases with increasing hole doping p. 
For p > 0.06 a magnetic structure with wave vector Q = (vr — 5, vr) (the corresponding mag- 
netic phase is referred as parallel below) replaces the diagonal incommensurate structure 
(Ref. S being approximately proportional to the hole doping up to p ~ 0.12^. At the 
same time, for the compound YBa2Cu306+y there exists a rather wide doping window in 
the vicinity of half-filling, where commensurate antiferromagnetism (or low-energy commen- 
surate antiferromagnetic fluctuations) is observed at low temperatures, probably related to 
the double-layer structure of this compound^'^'^'^. 

Cuprates are not the unique example of quasi 2D (layered) systems changing their mag- 
netic properties with varying physical parameters. The layered ruthenate Sr2Ru04 has two 



sheets o: 
7 (Ref. 



Fermi surface a,f3 formed by pairs of perpendicular planes and a cylindrical sheet 



lOl). The nesting provided by the a, (3 sheets causes low-energy magnetic fluctua- 



tions with wave vector Q = (O.Gtt, O.Gtt) carrying the dominant contribution to the magnetic 
spectral weight, but the 7 sheet also invokes low-energy fluctuations of moderate intensity 
with diagonal wave vector Q = (O.Stt, O.Stt)^^. The compound Sr2Ru04, when doped by La, 
acquires a strong tendency to ferromagnetic ordering (Ref. 1^, which is manifested itself by 
an enhancement of uniform susceptibility. However, no long-range ferromagnetic order is ob- 
served even for the Fermi level lying in the vicinity of the van Hove singularity. At the same 
time, the isostructural compound Ca2Ru04 exhibits ferromagnetism under pressure^i. Thus 
the magnetic phase structure of the ruthenates is very sensitive to experimental conditions. 
The properties of layered and three-dimensional interacting electronic systems are typ- 
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ically described within the Hubbard modeL Despite its simphcity this model allows, in 
particular, to explain rather complex phenomena originating from strong electron-electron 
interaction (ferromagnetism, antiferromagnetism and superconductivity) depending on the 
hopping parameters and electronic density. Surprisingly that although the Hubbard model 
has been studied for a long time, its magnetic phase diagram is not yet fully constructed even 
in the framework of the mean field (MF) approximation. Traditionally, only the competition 
of (collinear) ferromagnetic (FM) and antiferromagnetic (AF) phases was considered (see, 
e. g., Ref. I14J ). While the antiferromagnetism is energetically favorable at half-filling in the 
regime of strong electronic correlations, Nagaokar^ has argued that for three-dimensional 
Hubbard model at small doping and infinitely large electronic on-site Coulomb interaction 
U the ground state is saturated-ferromagnetically ordered; with decreasing U the ferromag- 
netic state appears to be unstable. 

Furthermore, Khomskii considered the possibility of a canted magnetic state at finite U, 
which is a superposition of ferro- and antiferromagnetic order—. Such a state turns out 
to be, however, energetically unfavorable because of its instability with respect to phase 
separation (PS). In particular, at large U Visscher— obtained the existence of PS region 
of ferromagnetic and antiferromagnetic states on the phase diagram and determined its 
location. The PS of these states was later extensively considered in Refs. Il8|]l9|]20l . It was 
showni^i^ that it is energetically favorable for conduction electrons to be localized in some 
ferromagnetic regions on the background of antiferromagnetically ordered local moments. 
Such autolocalized states have been called ferrons^^. Thus, the system gains in kinetic energy 
of electrons at the expense of loss in the local-moment subsystem energy. 

Another type of instability of paramagnetic (PM) phase is the formation of incommen- 
surate (spiral) magnetic structures in the ground state, which was studied in the last two 
decades for 2D itinerant systems^^i^SiSiiS^iS^i^i. Using the MF approximation, Arrigoni and 
Strinati considered the competition of diagonal and parallel spiral phases in the Hubbard 
model with nearest neighbor hopping and found a first-order transition between them^. 
Applying the Maxwell construction, these authors revealed the existence of spatial mixture 
of commensurate antiferromagnetic and incommensurate magnetic phases in some region 
of the phase diagram. Chubukov and Musaelian^^ considered the effect of small doping on 
the magnetic structure of the Hubbard model with finite next-nearest neighbor hopping and 
found that the diagonal spiral magnetic structure is unstable with respect to PS, while the 
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parallel phase is stable. Recently similar results were obtained within the t — J model at 
small doping^'^^. 

The electronic correlations were investigated beyond the standard MF approximation 
within the functional renormalization group (fRG) approach for small U, and dynamical 
mean field theory (DMFT) for intermediate and large U. The account of electronic correla- 
tions within the fRG approach of the 2D Hubbard model with nearest (t) and next-nearest 
(t') neighbor hopping in the vicinity of van Hove fdlin^^^^^^ showed that small val- 
ues of t'/t favor the competition between antiferromagnetic and d-wave superconducting 
ordering2^i^'^'^'2^. The incommensurate magnetic order was also shown to compete with 
both instabilities^'^. Within the DMFT the magnetic phase diagram for the Hubbard model 
was constructed on a Bethe lattice with nearest^S and next-nearest neighbor hopping. For 
small t' the PS of antiferromagnetic and paramagnetic phases, as well as an incommensurate 
magnetic phase (the region where commensurate solutions of the DMFT equations do not 
exist), were found. 

The ferromagnetic state was shown to compete with other instabilities in the 2D Hub- 
bard model for large t'/t (Refs. 31 S^QJsgI), being stable at low and moderate electronic 
densities^^'^''. Recently it has been shown that consideration of incommensurate magnetic 
fluctuations with small wave vectors significantly changes the boundary of ferromagnetic 
region on the phase diagram of 2D Hubbard model-S, since this region is "forced" out by 
diagonal incommensurate order. Therefore, the study of competition between ferromagnetic 
and incommensurate order acquires special importance in determining the conditions for the 
stability of the FM phase. Taking into account incommensurate fluctuations and the shift of 
the chemical potential within the quasistatic approximation reveals that even at van Hove 
filling, ferromagnetism cannot be realized at arbitrarily small U^^. The fRG calculations 
taking into consideration the electronic self-energy corrections^ also show the importance 
of incommensurate fluctuations near the ferromagnetic ground state. 

The discussed variety of magnetic orders is expected to be strongly influenced by PS 
phenomenon. In previous studies this influence was not considered systematically. The aim 
of the present paper is to relate the possibilitity of phase separation to incommensurate order, 
and to construct the magnetic ground-state phase diagram of the 2D Hubbard model on a 
square lattice with nearest and next-nearest neighbor hopping within the MF approximation, 
including both possibilities. This problem is relevant in the context of high-temperature 
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superconductivity in cuprates as well as for ruthenate systems. The plan of the paper is 
the following. In Sect. [Ill we consider possible types of magnetic ground states, present the 
derivation of MF approximation for spiral magnetic states and treat the formal aspects of 
the phase separation problem. In Sect. IIIII we present the results of our phase diagram 
calculations. Section [IV] is devoted to the discussion of the results. 

II. FORMALISM 

We consider the Hamiltonian of the Hubbard model on the square lattice 

^ = 5Z ^i^4aCja + C+Ci^cl^di, (l) 

ija i 

where tij = —t for the nearest-neighbor sites i,j and tij = t' for the next-nearest neighbors, 
c^(cjo-) is a creation (annihilation) electronic operator on site i with the spin projection a, U 
is the electronic (Hubbard) on-site interaction. Fourier transformation of the hopping term 
yields the electronic spectrum 

£k = — 2t(cos kx + cos ky) + 4t'(cos k^ cos ky + 1), (2) 

where k = (fc^., fc^), the lattice constant is taken equal to unity. The interacting part of the 
Hamiltonian ([T]) can be represented in the form 

Hint = UY. c^^c^TcjQi = U J](n2/4 - (m,u,)2), (3) 

i i 

where Uj is the (arbitrary chosen) i-dependent unit vector and we introduce the site density 
rii = J2a '^i-'^io- site magnetization rrij = | Xlo-o-' (^ta^aa'Cia operators. 

To demonstrate the peculiarities of competition between different magnetic states we 
first consider the limit of large U. In this limit spatial separation into antiferromagnetically 
ordered regions with one electron per site and ferromagnetically ordered hole-rich regions is 
most preferable. To obtain the boundary of the PS region we modify Visscher's arguments^^ 
for the case of square lattice with nonzero next-nearest neighbor hopping (in the derivation 
below we consider the case of electronic density n < 1). The saturated ferromagnetic phase 
has the energy 

^FM = Yl ^k/(£k) = -4tiV;,(l - (1 + 2t'/t)7inh/2), (4) 

k 
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where Uh is the hole density in the ferromagnetic region, and Nh is the total number of holes, 
f{e) = 0{fi — e) is the Fermi function at zero temperature {fi is the chemical potential, 6 
is the Heaviside step function. Note that actually depends on the number of antiferro- 
magnetically ordered sites A^af '■ f^h = Nh/{N — A^af)? N being the number of sites. The 
energy of the AF region is Ep^p = —At^Nxe/U. Minimizing the total energy E-pm + -E'af with 
respect to A^af, we obtain the equation for the PS region boundary (determined from the 
condition N^y = 0, so that Uh = 1 — n, where n is the electronic density) 

t/Ups{n) = (1 + 2sign(l - n)t'/t)TT{l - nf /2, (5) 

where we include the generalization to the case of n > 1. Therefore, for a given density n and 
U < Ups{n) the homogeneous magnetic state is unstable with respect to PS of ferromagnetic 
and antiferromagnetic states. However, for moderate values of U incommensurate spiral 
magnetic states are expected to be important, and we have to account also for them. 

To discuss the possibility of incommensurate order we apply the MF treatment of the 
interaction term ([3]) as foUows^^ 

Hint ^ Yl (-^^^ - h^™' - ^(^^) V4 + f/(m,u,)') . (6) 

i 

where ^ = —Un/2. We choose Uj directed along (nij) and assume that (rii) = n. The first 
term in Eq. ([6]) corresponds to the uniform charge mean field ^ (which can be interpreted as 
a shift of the chemical potential), the second term is the correction due to the site dependent 
mean magnetic field hj = 2U (rrij), and the other terms correspond to the shift of the total 
energy. 

We consider the spiral type of incommensurate magnetic order, which is a superposition 
of the rotation of order parameter in the xy-plane, modulated with some wave vector Q, 
and the ferromagnetic component perpendicular to the a;|/-plane^ 

(nij) = m(x sin?/' cos(QRj) + y sin '?/^sin(QRj) + zcos '?/'). (7) 

This generalizes the Khomskii's idea^^ of superposition of the FM and AF ordering in the 
vicinity of half-filling. Note that in this case only the direction, and not the magnitude 
of the magnetic moment, depends on the site number. This state can be contrasted to 
the collinear incommensurate spin-density wave stated, where only the magnitude, not the 
direction, depends on the site number (irij oc zcos(QRj)). We restrict our consideration to 
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the spiral type of magnetic order, bearing in mind the intuitive argument that one gains 
more magnetic energy for the largest possible magnetization at every site. It is obvious that 
the MF thermodynamical potential (TP) Qmf = — TlnTr [exp(HMF — /uA/")] (where is 
the MF approximation for the Hubbard Hamiltonian ([T]), A/" is the particle number operator, 
T is the temperature) can be expressed through the TP of non-interacting electrons in the 
self-consistently determined charge and magnetic fields. It is convenient to write down the 
parameter dependence of TP explicitly 

^^mf(Q, ^, m; ii)/N = rioiQ, ^, 2Um; n, -Un/2)/N - Un^/A + f/m^ (8) 

where VLq is the TP of non-interacting electrons in external charge and magnetic fields, 
calculated in the Appendix. 

We find the preferable magnetic phase (i.e. '?/',Q,m, n) by minimizing I^mf (El) with 
respect to these variables for a given chemical potential /i. We denote the values which 
provide this minimum as ^/'min(/^)5 Qmin(/^)5 "^min(/^)5 '"'min(/^)- For a given magnetic structure 
specified by Q,^/',/i, the density n and magnetization m satisfy the MF equations 

^ = ^ E(/(^kQ(e, h)) + /(E+Q(e, h))l (9) 

k 

^ = ^ E(/(^kQ(^' ^)) - /(^kQ(^' h))) co^{2e^{h) - ^), (10) 

k 

where h = 2Um, 9\^{h) and E^qii, h) are determined in the Appendix; the preferred values 
of Q, ip can be then obtained from the minimization of TP. 

In practice we determine the functions Qmm(/^)5 '^mm(/^)5 '^mm(yu) by using a rather dense 
grid for variables Q and ip to provide the minimum of f^MF numerically. (In fact we always 
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vanishes. 



have ^/'min(/^) = 7r/2, so that the 'ferromagnetic' component introduced in Ref. 
therefore in the following we omit ip). It is necessary to keep in mind that we always have to 
account for the possibility of a paramagnetic solution with m = (the solution with m 7^ 0, 
as a rule, is unique). Note that we choose the chemical potential /i as a basic variable instead 
of the density n, since this allows us to avoid technical problems connected with using the 
Maxwell construction for first-order transitions (see below) . Change to the n-dependence is 
easily given by the solution of MF equations. 

Since the magnetization m of the spiral phase can be expressed through /i and Q, exclud- 
ing m we obtain the TP as a function of Q, so that the wave vector Q serves as an order 



parameter specifying the type of magnetic ordering, except for the transition to the PM 
phase, where the magnetic phase wave vector cannot be specified. We classify spiral phases 
by the symmetry of the wave vector Q : Q = (0, 0) for the ferromagnetic phase, (vr, vr) for 
the antiferromagnetic (Neel) phase, (0, vr), (Q, tt) for the parallel spiral phases, {Q,Q) for 
the diagonal spiral phase; we also consider the spiral magnetic phase with the wave vector 
Q = {0,Q) (we assume due to the symmetry x ^ y that Qx < Qy), where Q takes an 
arbitrary value in the range < Q < tt. For the PM phase m = and Q is not fixed. 

We have a first-order phase transition, when the minimum of TP ([8]) for a given n 
is provided by two pairs of Q and m: (Qi,mi) and (Q2,''^2)- The transition through 
this point results in a jump in the magnetic structure parameters and a heterogeneous 
state appears. Since the solution of the MF equations gives the density as a function 
of the chemical potential and wave vector Q, n = n(yU, Q), this transition leads to a 
jump An(/i) = Qi,mi) — '^-(/i, Q2, ""^2)! as well. When n is between n(/i, Qi,mi) 

and n{fi,Q2,m2), the system consists of two spatially separated phases with densities 
?T,(/i, Qi, mi) and Q2, ^2) in a volume proportion, which provides an average density 
equal to n. If n is used as a basic variable, this result can also be obtained through the 
Maxwell construction, since ^^mf as a function of n is not convex and dfi/dn is not positively 
defined in this case. Using /i as a basic variable is technically much simpler and reproduces 
the Maxwell rule results. 



III. PHASE DIAGRAM 



We have performed numerical calculations comparing TP of different magnetic phases 
and varying fi and U for several ratios of t'/t = 0,0.2,0.45, solving the equations fl9l). f|T0l) . 
The results for different t'/t are presented below. 



A. t' = 



For t' = we have the particle-hole symmetry {n ^ 2 — n) and we restrict ourselves to 
the region < n < 1. 

The magnetic phase diagram using n as a basic variable (without restricting TP f^MF to be 



convex) is presented in Fig. la (see also Refs. l26|]27l ). We have the Neel antiferromagnetic 
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FIG. 1: (Color online) a) Magnetic phase diagram for t' /t = constructed using n as a basic 
variable, red solid lines denote the boundary between different phases, blue dashed lines denote 
boundary between phases obtained using as a basic variable (see Fig. 2 below), b) Density 
dependence of the x-component Qx {Qy = Qx) of the wave vector Q (left axis), chemical potential 
fi (right axis), magnetization m (left axis) for t' = 0,17/ 1 = 15. Solid lines represent the result 
obtained using /i as a basic variable, dashed lines correspond to using n as a basic variable. The 
plateau of the dependence fj.{n) corresponds to the PS region. Dashed line in the PS region 
corresponds to the 'virtual' diagonal spiral phase, unstable with respect to PS 

state only at half-filling (n = 1), in the vicinity of half- filling we observe only the spiral 
{Q, Q) (diagonal) phase, far away from half-filling we have a FM phase for large U/t and the 
parallel spiral phase ((0,7r) or {Q,n)) for moderate U/t. However, detailed consideration of 
the dependence of the chemical potential on density (see, e.g.. Fig. lb for U = 15t) reveals 
the insufficiency of this approach. The dependence fi{n) obtained using n as a basic variable 
(dashed line) has a negative slope in the vicinity of half-filling, hence the diagonal phase is 
unstable with respect to PS and the FM— > AF transition occurs through PS region. The 
use of as a basic variable (solid lines) treats correctly this instability: the plateau of fi{n) 
determines the position of of PS region 0.78 < n < 1. 

In Fig. 2 we present the magnetic phase diagram constructed using /i as a basic variable. 
As discussed above, the states in the vicinity of half-filling are phase separated: there is a 
separation of AF and {Q, n) phases for U/t < 8.5, AF and {Q, Q) phases for 8.5 ^ f^/i ^ H, 
and AF and FM phases for U/t > 11, the boundary lines between these regions are actually 
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FIG. 2: (Color online) Magnetic phase diagram for t' /t = obtained using as a basic variable, 
blue bold lines denote the second-order phase transitions, red solid lines the first-order phase 
transitions, blue dashed lines denote first-order phase transitions calculated without regard for PS 
(see Fig. la). The PS regions lie between two red solid lines, they are shaded and denoted as 
'Phi + Ph2' for the PS of phases Phi and Ph2. The large-C/ result for the boundary of the PS 
region of FM and AF phases ([5]) is plotted as a green (dash-dotted) line. Red dashed horizontal 
lines denote crossovers between two different PS regions 

crossover lines. In comparison with Fig. la, the pure diagonal phase is strongly forced out by 
the PS regions and shrunken into a small spot. The other regions are not strongly affected: 
only narrow PS regions are present far away from half-filling. To compare Visscher's result 
([5]) with the results of MP approach, we also plot in Pig. 2a the boundary line of the PS 
region of PM and AP phases in the limit of large U/t, Eq. ([5]). One can see that this 
result agrees well with the mean-field results. The paramagnetic region never takes part in 
PS as it was rigorously proven in a recent study for the t' = case^. On the other hand, 
approaches not considering the possibility of incommensurate magnetic order yield often the 
PS of paramagnetic and magnetic states^^i^S, contradicting thus to the rigorous results. 



B. t'/t = 0.2 

Por t' 7^ 0, the particle-hole symmetry is not preserved. The magnetic phase diagram for 
t'/t = 0.2 is presented in Pig. 3a. Comparing Pigs. 2 and 3a, we conclude that already for 
this value of t'/t strong asymmetry of the hole-doped (n < 1) and electron-doped (n > 1) 
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FIG. 3: (Color online) a) Magnetic phase diagram for t'/t = 0.2 using /i as a basic variable, 
notations are the same as in Fig. 2. b) Density dependence of the x-component Qx of wave 
vector Q (left axis, Qy = vr), chemical potential // (right axis), magnetization m (left axis) for 
t' /t = 0.2, U /t = 4. Notations are the same as in Fig. lb. In the inset we plot the function Qx{n) 
in the vicinity of half-filling 

sides is observed. In contrast to the t' = case we have a finite critical U / 1 for the AF state 
at half-filling2^i^i^. At the same time, with increasing t'/t the FM and diagonal {Q,Q) 
phase regions force out those of parallel phase for n < 1. On the electron-doped side a 
rather wide region of pure AF state is observed for U/t < 7 near half-filling. For larger U/t 
we have a PS region of AF and parallel incommensurate phases {{Q, vr) for U/t & [3; 8] and 
(0, tt) for U/t > 8). These regions are more extended along the f/-axis as compared to the 
t' = case (see Fig.2a). We find a ferromagnetic region on the electron-doped side only at 
very large U/t ^ 16. 



The hole doped side of the phase diagram was considered in Ref. |25| for small t'/t using n 
as a basic variable and the existence of {Q, it) phase at small U and {Q, Q) phase for larger 
U was found in the vicinity of half-filling. Contrary to the present results, the [Q, tt) phase 
was found to be stable with respect to PS in Ref. l25|. However, the doping dependence of 
the order parameter m was not taken into account in that study assuming that the top of the 
lower AF band Eb = —Um, considered as a reference point for fi, is fixed with doping. Our 
numerical calculations using n as a basic variable (see Fig. 3b, the plot for m(n)) reveal a 
strong dependence m{n), such that Eb has a negative slope for n < 1. Therefore the chemical 
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potential acquires the correction decreasing with increasing n, which makes the parallel 
phase near half-filling also unstable with respect to PS. 

The obtained results agree with the experimental data on doping dependence of the 
magnetic structure in the hole doped compound La2_pSrpCu04 which has a similar value of 
t' /t. The PS near half-filling may explain the fact that chemical potential almost does not 
depend on doping for < p < 0.1^^. Apart from that, the experimentally observed sequence 
of magnetic transitions AF^ (Q, Q) —>■ {Q, tt) with increasing doping pM'5. jg ^j-^e same as 
calculated for f//t ~ 4 (see details in Fig. 3b). Note that the calculated phase transitions 
pass through the PS regions. In the inset of Fig. 3b we also plot the dependence Qx{n) to 
illustrate the nearly linear relation between incommensurability and doping in the vicinity 
of PS region at small doping. The existence of PS of magnetically ordered and non-magnetic 
metallic phase was also recently observed in La2-xSr^Cui_yNiy04 compound^. 

Quite a different situation is observed for the electron- doped compound Ni2_xCe2;Cu04 
where pure AF phase extends up to electron doping x ~ 0.14^ in agreement with the phase 
diagram of Fig. 3a. Apart from that, a strong dependence of the chemical potential on 
doping is observed suggesting the absence of PS for this compound^''. This agrees with 
our results on the magnetic structure of the electron-doped side (see Fig. 3a) for moderate 
U/t ~ 4. Such small values of interaction U/t for cuprates may be explained by strong 
renormalization (screening) of the Coulomb interaction, and by the absorption of a large 
part of the electron-electron interaction into formation of Hubbard subbands. 

C. t'/t = 0.45 

Now we consider the phase diagram calculated for t'/t = 0.45, see Fig. 4a. One can 
see that below and slightly above the van Hove density {n = 0.46) a ferromagnetic ground 
state occurs, but well above the van Hove density a diagonal spiral phase with small wave 
vector is more preferable, the phase transition from the FM to diagonal spiral phase being 
of the second order. The details of such a transition are shown in Fig. 4b where the density 
dependences of the magnetic structure wave vector and chemical potential for U = At are 
presented. It is seen that a second-order transition between the FM and diagonal spiral 
phases is accompanied by a sharp drop of Q near transition. For n > 1 the parallel phases 
{Q, tt) and (0, vr) are dominating. We also have a pure AF phase in the vicinity of half-filling. 
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FIG. 4: (Color online) a) Magnetic phase diagram for \! jt = 0.45 using ^ as a basic variable, 
notations are the same as in Fig. 2; b) Density dependence of the x-component Qx of wave 
vector Q (left axis, Qy = Qx), chemical potential fi (right axis), magnetization m (left axis) for 
t' /t = 0.45, U/t = A. Notations are the same as in Fig. lb. 

The existence of extended FM and incommensurate regions on the hole-doped side of the 
phase diagram can be explained as follows. In the considered case of large enough t'/t the 
bottom of the band lies in the vicinity of the van Hove singularity. The general condition for 
ferromagnetic ordering, large DOS at the Fermi level, is easier fulfilled for densities below 
the van Hove filling. At the same time, spiral phases with small wave vectors compete 
with FM for fillings above van Hove filling because of the peculiarity of the momentum 
dependence of noninteracting magnetic susceptibility x{^,^ = 0); which has a maximum 
at q ^ 0. This competition was previously considered within the quasistatic approach in 
Ref. |38|, where the diagonal incommensurate magnetic phase was found to be the most 
significant for competition with the ferromagnetic phase. The critical value of U for the 
stability of ferromagnetism was shown to increase strongly due to this competition. Recent 
fRG calculations with self-energy corrections also suggest the existence of such a boundary 
which is close to that obtained above in terms of renormalized hopping parameters^. 

The obtained results may explain the magnetic properties of unconventional supercon- 
ductor Sr2Ru04 having four electrons per three bands crossing the Fermi level (Refs. Il2ljl3l). 
The contribution of the 7 sheet of Fermi surface, which is responsible for the tendency to 
diagonal incommensurate magnetic ordering, can be described by the one-band Hubbard 
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model assuming t' /t = —0.405 and n ~ 4/3 (Ref. |48|). This value of t' /t is close to that 
considered above, the difference in sign being absorbed by the wave function transformation 
(— which causes t' —t',n — > 2 — n, so that t'/t = 0.4, n ~ 2/3. If we use the 
renormalized f//t ~ 2 (Ref. 4^ we obtain a PM phase in the proximity of the transition to 
the diagonal incommensurate phase. With increasing U/t or decreasing density the latter 
phase undergoes the transition to the FM state. 

For all considered values of t'/t a common feature of the phase diagrams is the strong 
influence of PS on the magnetic structure in the vicinity of half-filling. We find a good 
agreement between the large-?/ result ([5]) and our result for the PS of FM and AF phases 
in the vicinity of half-filling at U/t > 12 and n < 1. Note that the result of Eq. for 
the hole-doped side of the phase diagram for large t'/t is much lower than that for the 
electron-doped side (not plotted), which qualitatively agrees with the absence of FM on the 
electron- doped side. 

IV. DISCUSSION AND CONCLUSIONS 

We have considered the ground-state magnetic phase diagram of the 2D Hubbard model 
with nearest (t) and next-nearest neighbor hopping (t') in the framework of the MF ap- 
proximation (see Figs. 2, 3a, 4a). We have taken into account the possibility of both the 
incommensurate (spiral) order and the phase separation (PS). We compared the thermody- 
namical potentials of different magnetic phases to determine the most preferable phase. The 
resulting phase diagram is rich due to the presence of spiral magnetic phases and PS, which 
has a dramatic effect on the phase diagram in the vicinity of the half-filling. In general, 
the diagonal incommensurate phase, which in some previous studies^^i^i^ was found to be 
stable in the vicinity of half-filling for n < 1, is replaced to a great extent by PS regions of 
different other magnetic phases. In contrast to previous approaches'"^ and in accordance 
with the exact results^ it was found that the PM phase does not take part in the PS (but 
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271 ). Therefore, 



the magnetically ordered phases do, contrary to the results of Refs. 
the interplay of spiral magnetic states and PS phenomena is of crucial importance. 

The breaking of the particle-hole symmetry due to the finite next-nearest neighbor hop- 
ping term {t') makes the phase diagram strongly asymmetric with respect to half-filling 
{n 2 — n). At large U we have PS of collinear ferro- (FM) and antiferromagnetic (AF) 
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states which agrees with the analytical laige-U result for the PS boundary line f/ps(n). For 
n > 1 the boundary curve Ups{n) is much higher than for n < 1, and the FM region is 
replaced by regions of spiral phase for moderately large U/t. With increasing t'/t we ob- 
serve the tendency to ferromagnetic and diagonal spiral magnetic ordering for n < 1 and 
the tendency to parallel spiral ordering for n > 1. The phases in the vicinity of half- filling 
are fully unstable with respect to PS for n < 1, but the AF region is found to be stable for 
n > 1, provided that t' ^ 0. 

The theoretical issue concerning the PS of incommensurate magnetic states in the vicinity 
of half-filling can be related to the explanation of some features observed in the one-layer 
compound La2_pSrpCu04 (Ref. [s]), in particular, the unusual dependence of the chemi- 
cal potential on hole doping^. Our approach can be generalized to the Hubbard model 
with bonding and antibonding bands, in order to model the electronic structure of double- 
layered compounds, in particular, to explain the significantly different magnetic behavior of 
YBa2Cu306+y upon doping. We believe the PS phenomena should be taken into account 
in more complicated approaches which are used for strongly correlated electronic systems, 
e. g., cuprates. 

Although the MF based approach provides a basic picture of magnetic ordering in the 2D 
Hubbard model, it does not take into account the effect of fluctuations, and its application 
to explaining the magnetic behavior of cuprates should be performed with caution. One 
should keep in mind, however, that for the first-order phase transitions fluctuations are 
not expected to change the obtained types of phases and to shift substantially the phase 
boundaries. At the same time, the phase transitions from magnetic to paramagnetic phase 
can be influenced more strongly by fluctuations. 

Trying to explain the PS phenomena in real compounds, it is impossible to avoid a consid- 
eration of additional long-range Coulomb energy originating from electronic inhomogeneity 
that is not taken into account within the Hubbard model. Rigorously speaking, the Hub- 
bard model is applicable only provided that the dopant-site distribution coincides with the 
electronic inhomogeneity distribution, thereby canceling the long-range Coulomb energy. If 
this cancellation is not perfect, the long-range Coulomb energy, as well as the surface energy 
of PS regions should be considered. The problem of PS in realistic systems requires therefore 
further consideration. 
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APPENDIX. CALCULATION OF THERMODYNAMICAL POTENTIAL OF 
NON-INTERACTING ELECTRONIC SYSTEM IN EXTERNAL CHARGE AND 
MAGNETIC FIELDS 

In this Appendix we solve an auxiliary problem of calculating the thermodynamical po- 
tential of non-interacting electrons in a uniform charge and a Q-modulated magnetic fields. 
The electronic Hamiltonian in the external charge, ^, and magnetic, /i, fields has the form 

= ^'i^taCja + ^n^,h, (11) 

ija 

where the correction due to the external fields ATC^^h is determined as 

AH^^h = - 5^(h.mi + en,), (12) 

i 

the operators rrij and are defined in the main text (see Sect. [Tll)- The magnetic field 
is assumed to depend on the site number as = /;,(xsin'?/' cos(QRi) + y sin sin(QRj) + 
zcos?/'), which yields 



sin^ ^(4+Q^Ckt + cj^t^k+Qi) + cost/; ^ o^^^Cka 



ko- 



eE^-^k., (13) 

kcr 



where we define the electronic Fourier transform as Qo- = Xlk ^^^^''^ko-- Therefore, we 
obtain 

= ^ka^Ck^ - {h/2) sin V; ^(4+Q^Ckt + ^^Ck+qJ. (14) 

kcr k 

with ^ko- = £^k — {h/2) cosipa — ^. Since the subspace spanned by the states (k t) and 
(k + Q I) (we denote it by W(k t, k + Q |)) is invariant with respect to H for any k (this 
means that H^^hl^ik t,k + Q |) C W(k t,k + Q |)), we can consider all these subspaces 
separately and the total matrix of the Hamiltonian is split into a direct sum of the 2x2 
matrices. Therefore we can restrict ourselves to the diagonalization of the 2x2 matrices. 

16 



We proceed with diagonalizing Ti^^^h (see Eq. (IHD) using the Bogolubov transformation. 
The magnetic field mixes the states (k |) and (k + Q |), so that the transformation reads 

Ckt = cos 6'kQakQ + sin 6'kQ/?kQ, Ck+Qj = - sin 6'kQakQ + cos 6'kQ/9kQ, (15) 

where tan(26'kQ(/i)) = /i sin (^kt — £^k+Qi)- After the transformation we obtain 

'>ii,h = Y.^Et^^i^^^Ci + i^kQ/^iq/^kQ), (16) 

k 

where the electronic excitation spectrum reads 

p± £kT + £k+Qi ± sign(ekT - ^k+Qi) V (^kT - £k+Qi)^ + sin^ i) 

Kq = 2 ■ ^ 

For density n and magnetization m we obtain 

^ = ^ 5Z(<"kQ"kQ) + {^3lct^3^.Q)), (18) 

k 

= ^ E(<^kQ/5kQ) - (4Q«kQ)) COs(2^^k(/^) - V'), (19) 
k 

where (alq^kq) = fiE^a), (/^i^q/^kQ) = /(i^^kq), /(e) = (1/2)(1 - tanh((£ - /i)/(2T)) is the 
Fermi function. For the TP we obtain 

l]o(Q,^,/^;/x,0 = -TlnTr[exp(-(7^5,,-/xAr)/r)] = J2iE^ci~fi)fiE^ci) + iE^Q-f^)fiE^Q)- 

(20) 

The right-hand side of Eq. (120]) . calculated at T = 0, is used in the main text, see Eq. 

dHD- 
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